Combining Textual and Visual Clusters for Semantic Image Retrieval and Auto-annotation

In this paper, we propose a novel strategy at an abstract level by combining textual and visual clustering results to retrieve images using semantic keywords and auto-annotate images based on similarity with existing keywords. Our main hypothesis is that images that fall in to the same textcluster can be described with common visual features of those images. In this approach, images are first clustered according to their text annotations using CM clustering technique. The images are also segmented into regions and then clustered based on low-level visual features using k-means clustering algorithm on the image regions. The feature vector of the images is then changed to a dimension equal to the number of visual clusters where each entry of the new feature vector signifies the contribution of the image to that visual cluster. Then a matrix is created for each textual cluster, where each row in the matrix is the new feature vector for the image in that textual cluster. A feature vector is also created for the query image and it is then appended to the matrix for each textual cluster and images in the textual cluster that give the highest coupling coefficient are considered for retrieval and annotations of the images in that textual cluster are considered as candidate annotations for the query image. Experiments have demonstrated that good accuracy of proposal and its high potential of use in annotation of images and for improvement of content based image retrieval.